EL5 80804246 



IN THE UNITED STATES PATENT AND TRADEMARK OFFICE 

APPLICATION FOR LETTERS PATENT 



An Improved Method and System for Rendering 

Concentric Mosaics 



Inventor(s): 
Heung-Yeung Shum 
Min-Sheng Wu 
Hong-Hui Sun 



ATTORNEY'S DOCKET NO. MS1-476US 



An Improved Method and system for Rendering Concentric Mosaics 



TECHNICAL FIELD 



The invention is related to an image-based rendering (IBR) system and method 
for rendering novel views of a 3D scene, including a stereo IBR system and method 
5 based on linear interpolated concentric mosaics. 



The ability to wander in a realistic 3D scene is attractive to a wide variety of 
people, ranging from scientists to kids. A significant problem to allowing such 

10 wandering is easy construction and real time rendering of complex 3D scene data. 

Traditional approaches often involve constructing or recovering a complete geometric 
and photometric model of the 3D scene by using modeling or vision techniques. Then, 
simulating the flow of light from a source, the light is reflected from the described model 
into a simulated camera and onto a film plane to produce an image. Such approaches are 

15 attractive as they allow various graphic elements, such as new objects, lights and 

shadows, to be readily added to the rendered scene. However, typically these approaches 
are also complex and computationally intensive. For most of the complex scenes, real 
time systems based on this approach are typically not able to run on high performance 
graphics workstations, much less typical desktop or portable computers. 
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BACKGROUND OF THE INVENTION 
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Another approach that is becoming increasingly popular is referred to as image 
based rendering (IBR). In IBR, the 3D scene is represented by a plenoptic function, such 
as is described by E.H. Adelson and J. Bergen in "The plenoptic function and the 
elements of early vision", Computational Models of Visual Processing, pp. 3-20, MIT 
5 Press, Cambridge, MA, 1991. The plenoptic function is constructed discretely by 
integrating images captured at a set of viewpoints and viewing directions. While 
rendering, novel views are generated by re-sampling the pre-constructed plenoptic 
function at given points, along certain paths. Unfortunately, the 7D plenoptic function 
described by Adelson and Bergen is typically too difficult to construct and store in 



;i] More recently, Heung-Yeung Shum and Li-Wei He in "Rendering with concentric 

rfl mosaics," Computer Graphics Proceedings, Annual Conference Series, pp. 299-306, 
□ Proc. SIGGRAPH'99, August 1999, describe an image rendering process based on 
§ .]f concentric mosaics. This image rendering process allows an observer to move freely 
«*15 within a circular region and allows images from the observer's viewpoint to be rendered 
in a more computationally efficient manner than many prior approaches. However, due 
to the manner in which images are rendered in the described technique, the quality of 
rendered images can sometimes suffer. Thus, it would be beneficial to provide a way to 
improve the quality of such rendered images. 
20 Another aspect of image rendering that has been approached is the rendering of 

stereo images. In S. Peleg and M. Ben-Ezra, "Stereo panorama with a single camera," 
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Proc. Computer Vision and Pattern Recognition Conference, 1999, stereo images are 
generated from a single rotating camera (off-centered). However, in the Peleg and Ben- 
Ezra system, the user is constrained to a fixed viewpoint - the scene cannot be wandered 

i 

through by the observer. Thus, it would be beneficial to provide a way to allow stereo 
5 rendering of images while wandering through a scene. 

The improved image rendering based on concentric mosaics described below 
addresses these and other disadvantages. 

3 SUMMARY 

01 10 Improved image rendering based on concentric mosaics is described herein. An 

y observer is able to wander through a real or synthesized scene and have new view images 

of the scene rendered as the observer moves. 
j= According to one aspect, a viewpoint and viewing direction are identified for an 

[I] observer. Multiple viewing rays are then extended in the viewing direction from the 
Q 1 5 image to be rendered at the view point. For viewing rays that coincide with captured 

images, the captured images are used for the rendered view. However, for viewing rays 
that do not coincide with captured images, an image is generated based on the captured 
images. The image is generated by interpolating between at least two captured images 
based on a constant distance to objects in the scene. 
20 According to another aspect, stereo images for the observer are rendered. Left 

and right eye images are generated that correspond to a pair of eyes of the observer. The 
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left and right eye images are generated based on the viewpoint and viewing direction, and 
are rendered substantially concurrently. Both images may be rendered on the same 
display, or alternatively on different displays (e.g., different screens of liquid crystal 
glasses). 

5 

DESCRIPTION OF THE DRAWINGS 

The present invention is illustrated by way of example and not limitation in the 
figures of the accompanying drawings. The same numbers are used throughout the 
figures to reference like components and/or features. 
m 1 0 Fig. 1 illustrates an exemplary system for rendering views of a 3D scene based on 

l\ j concentric mosaics in accordance with certain embodiments of the invention. 
W Fig. 2 shows a general example of a computer that can be used in accordance with 

certain embodiments of the invention. 
y& Fig. 3 illustrates an exemplary setup for capturing concentric mosaics, 

jh 15 Fig. 4 illustrates another exemplary setup for capturing concentric mosaics. 

Fig. 5 is a flowchart illustrating an exemplary process for rendering images in 
accordance with certain embodiments of the invention. 

Fig. 6 is a diagram illustrating an exemplary set of concentric circles and the 
concentric mosaics associated with two of them, as well as rays associated with a 
20 viewpoint that coincide with rays associated with the concentric mosaics in accordance 
with certain embodiments of the invention. 
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Fig. 7 is a diagram illustrating another exemplary set of concentric circles as well 
as a ray associated with a viewpoint in accordance with certain embodiments of the 
invention. 

Fig. 8 is a diagram illustrating an exemplary viewing ray and an interpolation 
5 process using constant depth in accordance with certain embodiments of the invention. 

Fig. 9 is a flowchart illustrating an exemplary process for rendering stereo images 
as an observer wanders about a scene in accordance with certain embodiments of the 
invention. 

Fig. 10 is a diagram illustrating exemplary viewpoints for stereo rendering in 
accordance with certain embodiments of the invention. 

Fig. 1 1 is a flowchart illustrating an exemplary process for generating stereo 
images in accordance with certain embodiments of the invention. 

DETAILED DESCRIPTION 

In the discussion below, embodiments of the invention will be described in the 
general context of computer-executable instructions, such as program modules, being 
executed by one or more conventional personal computers. Generally, program modules 
include routines, programs, objects, components, data structures, etc. that perform 
particular tasks or implement particular abstract data types. Moreover, those skilled in 
the art will appreciate that various embodiments of the invention may be practiced with 
other computer system configurations, including hand-held devices, gaming consoles, 

LEE & HAYES, PLLC 5 MSl-476US.pat.app.doc 

(o 





multiprocessor systems, microprocessor-based or programmable consumer electronics, 



network PCs, minicomputers, mainframe computers, and the like. In a distributed 



computer environment, program modules may be located in both local and remote 



memory storage devices. 



5 



Alternatively, embodiments of the invention can be implemented in hardware or a 



combination of hardware, software, and/or firmware. For example, all or part of the 



invention can be implemented in one or more application specific integrated circuits 



(ASICs). 



10 General System 

Fig. 1 illustrates an exemplary system for rendering views of a 3D scene based on 
hi concentric mosaics in accordance with certain embodiments of the invention. The system 
Q 100 can be implemented in software, hardware, firmware, or a combination thereof. A 
[U computing system 100 is illustrated including an operating system 102, an observer 
— 15 interface 104, a view Tenderer 106, an observer command input component 108, an 
observer display output component 110, and a scene data file 112. 

Operating system 102 performs various system level operations to control the 
lower level functions of the computing system and interaction of the various components 
104 - 1 12 of system 100. View renderer 106 generates the image seen by the observer as 
20 he or she wanders through the scene represented in scene data file 112 and outputs the 
image using display output component 110. View renderer 106 uses various parameters 
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to describe the observer's location and direction of view and uses the parameters, in 
conjunction with scene data file 1 12, to generate the image to output to the user. The 
geometric equations and relationships used to generate these parameters and apply them 
to scene data file 1 12 is discussed in more detail below. 
5 The user can move around within the 3D scene by inputting appropriate 

commands to observer interface 104 through command input component 108. These 
commands may correspond to input keystrokes on a keyboard, appropriate movements of 
a pointing device (such as a joystick, mouse or trackball, virtual reality glove, or other 
controller), voice based motion commands, etc. These commands are translated by 
m 10 observer interface 104 to a location within the 3D scene and a direction of view of the 
□3 observer that is presented for viewing by display output component 110. The display of 

M] output component 110 will change as if the user moved within the 3D scene. 

™ In addition to (or alternatively in place of) scene data file 112 being stored locally 

at system 100, a scene data file 1 14 may be accessed remotely (e.g., from a remote server 
f ] 15 116). System 100 can be coupled to server 1 16 by any of a wide variety of conventional 
communication channels 118, such as a local area network (LAN), wide area network 
(WAN), the Internet, direct dial-up connection, etc. Scene data file 114 may be 
transferred to memory of system 100 prior to rendering, or alternatively may be 
transferred on an on-going basis (e.g., as needed to render views). 
20 Fig. 2 shows a general example of a computer 142 that can be used in accordance 

with certain embodiments of the invention. Computer 142 represents, for example, 
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system 100 or server 116 of Fig. 1. Computer 142 includes one or more processors or 
processing units 144, a system memory 146, and a bus 148 that couples various system 
components including the system memory 146 to processors 144. The bus 148 represents 
one or more of any of several types of bus structures, including a memory bus or memory 
5 controller, a peripheral bus, an accelerated graphics port, and a processor or local bus 
using any of a variety of bus architectures. 

The system memory includes read only memory (ROM) 150 and random access 
memory (RAM) 152. A basic input/output system (BIOS) 154, containing the basic 
routines that help to transfer information between elements within computer 142, such as 
01 10 during start-up, is stored in ROM 150. Computer 142 further includes a hard disk drive 
^ 156 for reading from and writing to a hard disk, not shown, connected to bus 148 via a 

— ^ * 

hard disk driver interface 157 (e.g., a SCSI, ATA, or other type of interface); a magnetic 
j= disk drive 158 for reading from and writing to a removable magnetic disk 160, connected 
EO to bus 148 via a magnetic disk drive interface 161; and an optical disk drive 162 for 
Q 1 5 reading from or writing to a removable optical disk 1 64 such as a CD ROM, DVD, or 
other optical media, connected to bus 148 via an optical drive interface 165. The drives 
and their associated computer-readable media provide nonvolatile storage of computer 
readable instructions, data structures, program modules and other data for computer 142. 
Although the exemplary environment described herein employs a hard disk, a removable 
20 magnetic disk 160 and a removable optical disk 164, it should be appreciated by those 
skilled in the art that other types of computer readable media which can store data that is 
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accessible by a computer, such as magnetic cassettes, flash memory cards, digital video 
disks, random access memories (RAMs) read only memories (ROM), and the like, may 
also be used in the exemplary operating environment. 

A number of program modules may be stored on the hard disk, magnetic disk 160, 
5 optical disk 164, ROM 150, or RAM 152, including an operating system 170, one or 
more application programs 172, other program modules 174, and program data 176. 
Operating system 170 can be any of a variety of operating systems, such as any of the 
"Windows" family of operating systems available from Microsoft Corporation of 
Redmond, Washington. A user may enter commands and information into computer 142 

■ I J 

~ — *» 

* - ; 

%! 10 through input devices such as keyboard 178 and pointing device 180 (such as a mouse). 

* ™ * 

f jj Other input devices (not shown) may include a microphone, joystick, game pad, satellite 
tlj dish, scanner, motion tracking devices, virtual reality gloves, controllers, or the like. 
Q These and other input devices are connected to the processing unit 144 through an 
j^f interface 168 (e.g., a serial port, parallel port, game port, or a universal serial bus (USB)) 
15 that is coupled to the system bus. A monitor 184 or other type of display device (such as 

* * 

a head mounted display, a 3D video projection, or the like) is also connected to the 
system bus 148 via an interface, such as a video adapter 186. In addition to the monitor, 
personal computers typically include other peripheral output devices (not shown) such as 
speakers and printers. 

20 A stereo display device 185 and shutter crystal glasses 187 can also be included as 

an output device for the computer 142. Shutter crystal glasses 187 operate in conjunction 
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with video adapter 1 86 to present stereo image pairs to left and right eyes of a user. Any 
of a wide variety of commercially available crystal glasses can be used, such as 
CrystalEyes available from StereoGraphics Corporation of San Rafael, California. 
Alternatively, stereo image pairs can be output by stereo display device 185 and/or video 
adapter 186 to other display devices as well, such as monitor 184, a head-mounted 
display (not shown), etc. 

Computer 142 can optionally operate in a networked environment using logical 
connections to one or more remote computers, such as a remote computer 188. The 
remote computer 188 may be another personal computer, a server, a router, a network 
PC, a peer device or other common network node, and typically includes many or all of 
the elements described above relative to computer 142, although only a memory storage 
device 190 has been illustrated in Fig. 2. The logical connections depicted in Fig. 2 
include a local area network (LAN) 192 and a wide area network (WAN) 194. Such 
networking environments are commonplace in offices, enterprise-wide computer 
networks, intranets, and the Internet. In the described embodiment of the invention, 
remote computer 188 executes an Internet Web browser program such as the "Internet 
Explorer" Web browser manufactured and distributed by Microsoft Corporation of 
Redmond, Washington. 

When used in a LAN networking environment, computer 142 is connected to the 
local network 192 through a network interface or adapter 196. When used in a WAN 
networking environment, computer 142 typically includes a modem 198 or other means 
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for establishing communications over the wide area network 194, such as the Internet. 
The modem 198, which may be internal or external, is connected to the system bus 148 
via a serial port interface 168. In a networked environment, program modules depicted 
relative to the personal computer 142, or portions thereof, may be stored in the remote 
5 memory storage device. It will be appreciated that the network connections shown are 
exemplary and other means of establishing a communications link between the computers 
may be used. 

Generally, the data processors of computer 142 are programmed by means of 
instructions stored at different times in the various computer-readable storage media of 



U * 10 the computer. Programs and operating systems are typically distributed, for example, on 



floppy disks or CD-ROMs. From there, they are installed or loaded into the secondary 
memory of a computer. At execution, they are loaded at least partially into the 
computer's primary electronic memory. The invention described herein includes these 
and other various types of computer-readable storage media when such media contain 

15 instructions or programs for implementing the acts described below in conjunction with a 
microprocessor or other data processor. The invention also includes the computer itself 
when programmed according to the methods and techniques described herein. 
Furthermore, certain sub-components of the computer may be programmed to perform 
the functions and acts described herein. The invention includes such sub-components 

20 when they are programmed as described. In addition, the invention described herein 
includes data structures, described below, as embodied on various types of memory 
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media. 



For purposes of illustration, programs and other executable program components 
such as the operating system are illustrated herein as discrete blocks, although it is 
recognized that such programs and components reside at various times in different 
storage components of the computer, and are executed by the data processor(s) of the 
computer. 

Returning to Fig. 1, scene data file 112 and/or 1 14 are based on a set of mosaics. 
A mosaic is a collection of images of a surrounding 3D scene captured with camera 
motion constrained within a single plane. Multiple mosaics can be captured with the 
camera motion constrained to movement about a point of rotation within the same plane. 
A set of such multiple mosaics are referred to as concentric mosaics. 

Fig. 3 illustrates an exemplary setup for capturing concentric mosaics. A capture 



system 210 includes a beam 212 rotatably coupled to a support 214, allowing beam 212 
to be rotated substantially 360 degrees. The location of this coupling defines a center of 
rotation 216. Multiple cameras 218 are placed along beam 212. As beam 212 is rotated 
a, images are being captured by cameras 218 along the tangent direction of the circle of 
rotation. The images captured by each camera 218 can be slit images (typically one pixel 
wide), or alternatively "whole area" images (i.e., a wider angle of capture than the 
typically single-pixel wide slit images, such as a 90 degree or 180 degree field of 
capture). The cameras 218 can be of any type appropriate for capturing images of the 
scene, such as video cameras (e.g., digital or non-digital), still cameras (e.g., film or 
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digital), Charge Coupled Device (CCD) elements, etc. 

Fig. 4 illustrates another exemplary setup for capturing concentric mosaics. A 
capture system 230 includes a beam 232 rotatably coupled to a support 234, allowing 
beam 232 to be rotated substantially 360 degrees about a center of rotation 236. A 
camera 218 is placed on beam 232 and a counterweight 238 is optionally placed at the 
end of the beam opposite camera 218. As beam 232 is rotated, images are captured by 
camera 218 along the radial direction of the circle of rotation. 

The cameras 2 1 8 in Figs. 3 and 4 define a circular region as they rotate about the 
center of rotation. This circular region is referred to herein as a "capture circle". 

Alternatively, "capture" images can be generated synthetically, such as by using 
any of a wide variety of conventional graphics programs capable of producing a 3D 
scene. These programs can be employed to generate images of the desired dimensions, 
which depict a portion of a surrounding synthetic 3D scene from a perspective that is 
radial (or alternatively tangential) to a circle on a circle plane disposed within a synthetic 
3D scene. Accordingly, it is to be understood that the "captured images" referred to 
herein could refer to either real images or synthetic images. 

Image Rendering Using Linear Interpolation 

Since concentric mosaics have captured many of the rays in the capture plane 
within the circular region defined by the movement of the capture camera(s), viewing 
rays from within the circle can be rendered. This allows the observer to move 

LEE & HA YES, PLLC 13 MSl-476US.pat.app.doc 




continuously within or nearby the circular region to observe motion parallax and lighting 
changes. However, only a subset of all the rays in the plane need be stored, so, 
depending on the observer's viewing position and viewing direction, some rays may need 
to be approximated. In accordance with certain embodiments of the invention, linear 
5 interpolation with constant depth is used to approximate such rays. 

Fig. 5 is a flowchart illustrating an exemplary process for rendering images in 
accordance with certain embodiments of the invention. The process of Fig. 5 is 
implemented by, for example, a system 100 of Fig. 1, and may be performed in software. 
Fig. 5 is discussed with reference to concentric mosaics rendered based on portions that 
10 are slit images (e.g., one pixel wide) tangential to the direction of rotation of the cameras. 
The rendering can be based on, for example, slit images captured tangential to the 
direction of rotation of the camera(s) or whole images captured radial to the direction of 
rotation of the camera(s). If the images are captured radial to the direction of rotation of 
the camera(s), then a concentric mosaic is formed by combining the same vertical line 

th 

15 (e.g., the 20 vertical scan line) taken at different rotation angles. The concentric mosaic 

so formed is the same as a concentric mosaic formed by capturing slit images tangential 

to the direction of rotation. 

A view image from an observer's viewpoint inside the 2D circular region of the 

capture circle can be rendered by first identifying all the rays emanating from the 
20 viewpoint that represent the locations of the slit images used to construct the rendered 

view (act 260). Although these rays emanate from the same viewpoint, they emanate in 
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different viewing directions (encompassing a latitudinal field of view of the observer). 
These viewing directions are together represented by the direction of viewing of the 
observer (which is, for example, one of the viewing directions (e.g., the centermost 
viewing direction)). Next, one of the identified rays is selected and a check is made as to 
5 whether the selected ray coincides with a previously captured slit image (act 262). This 
can be accomplished by checking both the radial location and the rotation angle of the 
captured slit images. 

If there is such a coinciding ray, then the associated previously captured slit image 
associated with the coinciding ray is selected as one of the slit images used to form a part 



10 of the rendered view image (act 264). The selected ray is also associated with the 

coinciding ray emanating from the viewpoint of the rendered view image (act 266). This 
can be accomplished by associating a unique identifier for the slit image (e.g., its radial 
location and angle of rotation) with the appropriate ray emanating from the rendered 
view's viewpoint. 



the circle plane 300 with several of the concentric circles (C 0f C*, C/, and C n ) 302, 304, 
306, and 308. In addition, rays 310 and 312 are shown emanating from viewpoint (P) 
314 and respectively traversing the circle plane 300 toward two of the slit images needed 
to form the novel view. To render a novel view from viewpoint 314, the captured slit 
20 images that coincide with rays 310 and 312 are found. As all the captured slit images 
where taken in a direction tangent to one of the concentric circles, a coinciding captured 
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This process is illustrated schematically in Fig. 6. The left side of Fig. 6 shows 




slit image can be found by determining where a ray emanating from viewpoint 314 
becomes tangent with one of the concentric circles. In the depicted case, ray 310 is 
tangent to circle 304 at point (v,) 316 and ray 312 becomes tangent to circle 306 at point 
(vy) 318. Thus, the rays that are associated with these points 316 and 318, and have the 
5 same direction as rays 310 and 312, identify the slit images needed to form part of the 
view image. 

There may be, however, practical limits to the number of concentric mosaics that 
can be captured and/or stored. A limitation on the number of concentric mosaics can 
have an impact on the rendered view images because some of the ray images needed to 

10 form a rendered view image may not exactly line up with one of the captured slit images. 
In other words, it is possible that an image ray needed to form a part of the rendered view 
be approximated based on two or more captured rays. 

Referring to Fig. 7, the foregoing situation can be illustrated as a ray 340 
emanating from a viewpoint 342 associated with a view image, which never becomes 

15 tangent to any of the concentric circles 344, 346, and 348 on the circle plane 350. Instead 
the ray 340 passes between two of the circles 346 and 348. Therefore, no ray associated 
with a captured slit image will coincide with the ray 340 (hereinafter referred to as a 
"non-coinciding ray"). 

One way of dealing with this situation is via a linear interpolation process. Linear 

20 interpolation involves blending multiple captured slit images to form the slit image 
required for the rendered view image being created. The blending process assigns a 
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weight to each pixel display value in the multiple captured slit images and adds these 
values together proportionally. In one implementation, described in more detail below 
(with reference to Fig. 8), four slit images (based on four rays) are blended based on 
weighted values derived from using a constant or fixed depth. Alternatively, fewer or 
5 greater numbers of images may be used in the interpolation process. Returning to Fig. 5, 
this blending is accomplished by initially identifying four rays based on the non- 
coinciding ray and a constant depth, with each of the four rays coinciding with a captured 
slit image (act 268). 

A weight is then assigned to each of the four rays (act 270). The captured slit 

10 images coinciding with these four rays, and the assigned weights, are then used to 

interpolate a non-coinciding slit image (act 272). In one implementation, a computed 
ratio specifies the weight each of the rays (and thus the proportion of each of the captured 
slit images coinciding with the rays to be used in generating the pixel values for the 
interpolated slit image) is to be given in the blending process to generate the interpolated 

15 slit image needed to construct the rendered view image. 

Once the interpolated slit image is generated, it is associated with the appropriate 
ray emanating from the viewpoint of the rendered view image (i.e., the non-coinciding 
ray) (act 274). The above sequence of operations will continue until slit images for all 
the rays to be used for the rendered view image are generated (act 276). 

20 The rendered view image is then generated by combining all the identified and 

interpolated rays in the proper order to create the rendered view (act 278). Referring to 
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the right hand side of Fig. 6, the combining process is illustrated for the case where rays 
associated with captured slit images coincide with rays identifying slit images needed to 
construct the rendered view image. The slit image (Li) 320 of the concentric mosaic 
(CM*) 322 is the slit image identified by the captured ray associated with point (v,-) 316. 
Similarly, the slit image (Lj) 324 of the concentric mosaic (CM/) 326 is the slit image 
identified by the captured ray associated with point (vj) 318. 

Therefore, these slit images 320 and 324 can be used to construct part of the 
rendered view image 328. Specifically, slit image (Li) 320 is placed in the appropriate 
location, as slit image (/) 330, in the rendered view image 328. Similarly, slit image (Lj) 
324 is placed in the appropriate location, as slit image (/) 332, in the rendered view image 
328. In the case where a linear interpolation process is used to produce a slit image for 
the rendered view from captured slit images, the slit image placement is performed after 
the interpolated slit image is constructed. 

Bilinear interpolation with constant depth (also referred to as fixed depth) is one 
linear interpolation process used for interpolating in accordance with certain 
embodiments of the invention. While capturing concentric mosaics, the distance from the 
rotation center to a camera position r can be determined (e.g., the distance between center 
of rotation 216 and a camera 218 of Fig. 3) can be recorded. Additionally, the distance 
from the rotation center to the nearest object of interest (i? m i n ), as well as the distance 
from the rotation center to the farthest object of interest (i? m ax) can also be recorded. 
During interpolation, it is assumed that all the objects have a constant depth 
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^ = CKmin+^max)/2 (i.e., they are all located at a cylinder with radius 7?). Alternatively, the 
constant depth can be determined in other manners, such as a median depth based on all 
(or alternatively less than all) of the objects. 

The nearest and farthest objects of interest can be manually or automatically 
5 selected. These objects can be absolutes (e.g., the physically closest or physically 

farthest objects), representative of a group of objects that are at approximately the same 
distance, approximations, etc. 

Alternatively, rather than using the same depth R for all objects, the depth R can 
vary through different portions of the scene. These different portions can be defined, for 
10 example, by radial locations. By way of example, the first 90 degrees may have a first 
depth, the next 180 degrees may have a second depth, and the next 90 degrees may have 
a third depth. 

The bilinear interpolation process is discussed in additional detail with reference 
to Fig. 8. Given a viewpoint (P) 400 and a viewing direction, a line 402 (PQ) can be 



15 generated. The point 404 (Q) is the intersection point of the outermost capture circle and 
a viewing ray extending from viewpoint (P) 400 in the view direction. Alternatively, a 
different circle other than the outermost capture circle may be used for point 404 (0). 
Let line 402 (PQ) intersect with the cylinder having an origin (O) 406 and a radius R at 
point 408 (7), where \07]=R. 

20 Next, two sample points 410 (Qj) and 412 (Q 2 ) are determined, each 

corresponding to a camera location where an image was captured. These sample points 
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410 (Qj) and 412 (Q2) are closest to the point of intersection (Q) 404 along the capture 
circle 413 at a distance r from the origin (0). From each of the two sampled viewpoints 
410 and 412, a line to point 408 (7) is drawn, resulting in lines 414 (PjQi) and 416 
(P2Q2), respectively. For both lines 414 and 416, the two sample rays (each coinciding 
5 with a previously captured slit image) that are closest to each of them is determined, 
resulting in four sample rays (Li, L2, L3 and L 4 ) 418, 420, 422, and 424. Each of these 
four sample rays 418, 420, 422, and 424 is given a weight, and the weighted values are 
used to calculate values for the non-coinciding ray 402 (PQ). 



10 420, 422, and 424 are calculated as follows. The weight of sample ray 418 is w\ 9 is 
calculated as follows: 



where y } is the angle between lines 414 and 402, y 2 is the angle between lines 402 and- - . 
416, q>] is the angle between ray 418 and line 414, and q>2 is the angle between line 414 
15 and ray 420. 



In one implementation, the weights assigned to each of the four sample rays 418, 





The weight of sample ray 420 is W2, is calculated as follows: 
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where ji is the angle between lines 414 and 402, y 2 is the angle between lines 402 and 



416, cpi is the angle between ray 418 and line 414, and <p 2 is the angle between line 414 
and ray 420. 

The weight of sample ray 422 is w 3 , is calculated as follows: 



w 3 =- 

Yi+Yi <Pi+<P* 



where cps is the angle between ray 422 and line 416, and q> 4 is the angle between line 416 
and ray 424. 

^ The weight of sample ray 424 is w 4 , is calculated as follows: 



10 where q>3 is the angle between ray 422 and line 416, and <p 4 is the angle between line 416 
and ray 424. 

The values for yi and Y2 are calculated as follows. Initially, a value 0 is 
calculated, 

~? 



0 j /-x . (rsinfl^ 

/ r\ f\Jr-^ & = a- arcsin 

A 



v R j 



1 5 where a is the angle between the x axis and line 402, r is the distance between the 

rotation center and the capture circle 413, R is the constant depth calculated above, and J3 
is the angle between line 402 and a line 426 from the origin 406 to point 404. Given 0, 
the values of vectors T, Qi, and Q2 can be calculated as follows, 
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T = (tfcos0 9J Rsin0) 

Q, = (rcos0 15 rsin^,) 

Q 2 = (rcos# 2 , rsin# 2 ) 

Once the values of the vectors T, Qi, and Q2 are calculated, two values ai and oti can be 
calculated as follows, 

a x = arctan(T.j> - Q, .y 9 T.jc -Q v x) 

a 2 = arctanCT.jy - Q 2 .y, T.x - Q 2 .x) 
Given the two values ai and a^, the values for yi and Y2 can be calculated as follows, 

y 2 =a-a 2 

Stereo Image Rendering 

In certain embodiments of the invention, images can be rendered in stereo. Stereo 
rendering often better simulates the viewing of the surrounding scene by an observer as it 
can simulate the view from a pair of eyes. Stereo rendering is further improved when the 
rendering is performed with a stereo viewing device, such as the shutter crystal glasses 
described above. 

Fig. 9 is a flowchart illustrating an exemplary process for rendering stereo images 
as an observer wanders about a scene in accordance with certain embodiments of the 
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invention. The process of Fig. 9 is implemented by, for example, a system 100 of Fig. 1, 
and may be performed in software. 

Initially, the system is initialized and the data of the concentric mosaics (e.g., 
scene data file 1 12 or 1 14) is loaded into memory (act 500). This initialization can 
5 include a default viewing location and direction of viewing for the observer, so an initial 
rendered view image is generated. Alternatively, some or all of the data is not loaded 
into memory; rather, the data is loaded into memory later (e.g., when needed to generate 
a rendered view image). 



10 the circular regions defined by the set of concentric mosaics (act 502). The system 

receives the input information via various user interfaces, determines whether the input is 
legal for rendering stereo images (e.g., the movement does not exceed the capture circle 
or does not exceed the capture circle by more than a predetermined amount), and 
generates the necessary parameters needed to describe the observer's viewpoint and 



1 5 viewing direction(s). 

In response to each user movement input, stereo image pairs of the 3D scene (e.g., 
a left eye image and a right eye image) are rendered from the new viewpoint defined by 
the user input (act 504). The stereo image pairs represent, for example, the surrounding 
scene an observer would see if standing within the 3D scene captured by the concentric 

20 mosaics. This rendered view is then displayed into a stereo display device (act 506) for 
viewing by the user. 
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Once initialized, the system accepts user input information for movement within 




As shown in Fig. 10, a local coordinate system is used to describe the motion of 
an eye pair (left eye 520 and right eye 522). The local coordinate system 524 maps to a 
global coordinate system 526. The distance between the two eyes 520 and 522 is denoted 
by 21. The viewing positions of the left and right eyes 520 and 522 equal the position of 
5 the center of the connection line 528 between the eyes 520 and 522 minus and plus the 
value /, respectively. 

The eye pair 520, 522 possesses a total of 5 freedoms of motion (/, a, x, y, 0). 
The motion / is translation freedom that allows the user to adjust the distance 528 
between the eyes 520 and 522. The motion a is rotation freedom that allows the user to 



10 adjust the angle between the eyes 520 and 522 (e.g., to obtain intaglio or rilievi effects). 



The motions x and y are translation freedoms that allow the user to move freely within a 
2D circular region. The motion 9 is rotation freedom that allows the user to turn his or 
her head in any 2D direction. 



15 104 of Fig. 1), they are used (e.g., by view renderer 106 of Fig. 1) to render a 
corresponding stereo image pair. 

Fig. 1 1 is a flowchart illustrating an exemplary process for generating stereo 
images in accordance with certain embodiments of the invention. The process of Fig. 1 1 
is implemented by, for example, view renderer 106 of Fig. 1, and may be performed in 

20 software. 
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Once the viewpoint and eye parameters are generated (e.g., by observer interface 




Initially, a set of necessary parameters needed to define the viewpoint and 
viewing direction(s) for the left eye image are determined (act 540). Using the left eye 
view parameters, a first rendered view image is generated (act 542). Next, a set of 
necessary parameters needed to define the viewpoint and viewing direction(s) for the 
5 right eye image are determined (act 544). Using the right eye view parameters, a second 
rendered view image is generated (act 546). The two rendered view images are now able 
to be presented to a display component (e.g., display output component 110 of Fig. 1). 

In one implementation, the viewpoint of the eye pair is calculated from a center 
point between the two eyes. Alternatively, the view point may be calculated based on 
10 one or the other of the two eyes. The viewing direction(s) for both of the eyes is the same 
(or may be changed by adjusting the angle between the eyes). 



During rendering of each of the stereo images (in acts 542 and 546), non- 
coinciding rays can be encountered as discussed above with reference to Fig. 5. Values 
for such rays can be determined using the linear interpolation with constant depth as 
15 discussed above, or alternatively using other interpolation techniques. 

Conclusion 

Thus, improved image rendering based on concentric mosaics has been described. 
In certain embodiments, view images are rendered by calculating the values for portions 
20 of images corresponding to viewing rays from an observer's view point in a viewing 
direction(s). In the event that a viewing ray does not coincide with a captured image, 
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linear interpolation with constant depth is advantageously used to calculate a value for 
the portion of the image corresponding to the viewing ray. Additionally, in certain 
embodiments stereo rendering of images is performed, allowing a more realistic "two- 
eye" view of the surrounding scene. 
5 Although the description above uses language that is specific to structural features 

and/or methodological acts, it is to be understood that the invention defined in the 
appended claims is not limited to the specific features or acts described. Rather, the 
specific features and acts are disclosed as exemplary forms of implementing the 

)it invention. 

^ 10 



LEE & HAYES, FLLC 



26 



MSl-476US.pat.app.doc 



